Terminology

⚠ Terms/concepts to be careful with in ML, coming from stats:

• hypothesis (sometimes an output of a classifier model)

• bias (broader meaning)

• causality (sometimes less rigorous than stats)

Some we have already seen

• Simple linear regression
• Logistic regression
• Multiple linear regression

Model testing / validation

• ML gives us many ways to do the same tasks (regress, classify).
• With supervised methods, we can furthermore assess each model’s performance by comparing the prediction to observed output values. These comparisons are expressed with metrics such as \(R^2\).
• However, to be confident that we are not overfitting, we should split the data in some proportion (e.g., 80/20) prior to fitting. Then we can assess not only how the model fits the “training” (fitting) data, but also how it fits previously-unseen data, called “test” data
• Let’s split the penguins 80/20
• There are much more sophisticated ways of validation, such as fitting many models on subsets of the data and averaging their results

library(tidymodels) 

# setting seed ensures that "random" behavior is reproducible, i.e. the same every time
set.seed(1234)
penguins_clean <- penguins %>% 
  drop_na()
pens_split <- initial_split(penguins_clean, prop = 0.8)
pens_train <- training(pens_split)
pens_test <- testing(pens_split)

Naive Bayes classifier

• We assume that each predictor variable has a normal (Gaussian) distribution
  • For each variable, the distribution is estimated based on observed data
  • To classify an observation, plot it and check its probability of membership for each class.

Animation of the naive Bayes classifier. Color intensity indicates probability of group membership. Image source: Jacopo Bertolotti via Wikimedia Commons (CC0)

Naive Bayes examples

• Predicting water quality (Ilic et al. 2022)

Naive Bayes code example

library(naivebayes)

# predict sex from species and body mass:

sex_fit <- naive_bayes(sex ~ species + body_mass_g, data = pens_train)

library(klaR)
plot(sex_fit)
predict(sex_fit, data.frame(body_mass_g=3500, species="Gentoo"))

Animation of a simple decision tree example. Each binary branch in the tree on the left corresponds to a partitioning in the x-y space. The response variable (output) of this model is gray/green color classification. Image source: Algobeans

Image source: James et al. 2021

Decision tree code example

library(rpart)
library(rpart.plot)

pens_tree <- rpart(sex ~ species + body_mass_g, data = pens_train)x
rpart.plot(pens_tree)

K Nearest Neighbors

• A classifier
• “For my observation of interest, which class do the K nearest neighbors belong to?”
  • K = chosen by analyst
  • “Nearest neighbor” is defined using distance in the predictor space

K Nearest Neighbors

Effects of changing K.

K Nearest Neighbors code example

library(kknn)

# centering and scaling data:
pens_recipe <- recipe(sex ~ species + body_mass_g, data = pens_train) %>% 
  step_dummy(all_nominal(), -all_outcomes()) %>% 
  step_center(-sex) %>% 
  step_scale(-sex) %>% 
  prep()

pens_train_juiced <- juice(pens_recipe)

# visualize processed data:
pens_train_juiced %>% pivot_longer(-sex) %>% 
    ggplot() +
    geom_histogram(aes(value, fill = sex)) +
    facet_wrap(~name)

baked_test <- bake(pens_recipe, new_data = pens_test)

Fitting

knn_spec <- nearest_neighbor() %>% 
  set_engine("kknn") %>% 
  set_mode("classification")

knn_fit <- knn_spec %>% 
  fit(sex ~ ., 
      data = pens_train_juiced)

knn_fit

Metrics:

knn_fit %>% 
    predict(baked_test) %>% 
    bind_cols(baked_test) %>% 
    metrics(truth = sex, estimate = .pred_class)

Unsupervised learning

Unsupervised learning has no predictive model: instead it finds previously unknown structure in the data. All variables or features of the data are considered together.

Unsupervised learning tends to be most useful for exploratory data analysis, i.e., prior to having a goal for regression or classification.

• Principal Components Analysis (PCA): dimensionality reduction
• \(K\)-Means Clustering

Animation demonstrating projection of two features onto a single histogram using principal components analysis. Image source: Amélia O. F. da S. via Wikimedia Commons (CC BY-SA 4.0)

Image source: James et al. 2021

PCA examples

• Characterizing how people perceive themselves in a mirror versus looking at their own body (Jenkinson and Preston 2017)
• Diagnosis of diverse diseases from blood microRNA datasets (Sell et al. 2020)

PCA code example

PCA is a recipe step.

Read more here for important details: https://recipes.tidymodels.org/reference/step_pca.html

pca_recipe <- pens_recipe %>% 
  step_pca(all_numeric_predictors(), num_comp = 3) %>% 
  prep()

pca_juiced <- juice(pens_recipe)
pca_juiced

Animation of the K-means algorithm in action. After initial random group assignment, centroids are randomly placed and used to classify. Then centroids and assignment are iteratively adjusted until movement stops. Image source: Chire on Wikimedia Commons (CC BY-SA 4.0)

150 observations in 2D space, clustered according to different values of K. Prior to clustering, data are not categorized. Colors indicate which group each observation is assigned to by the model. Image source: James et al. 2021

K-Means examples

• Identifying patients at risk for paternal age-related schizophrenia (Lee et al. 2011)

K-Means code example

library(tidyclust)
kmeans_spec <- k_means(num_clusters = 4)
kmeans_spec

kmeans_fit <- kmeans_spec %>% 
  fit(~ body_mass_g + bill_length_mm, data = pens_train)

kmeans_fit %>% extract_fit_summary()
kmeans_fit %>% extract_cluster_assignment()

Hierarchical clustering

• Hierarchical clustering splits the dataset into nested groups, until all observations are in a cluster of 1
  • This splitting approach is divisive hierarchical clustering
• The result is a tree (dendogram), and we can assess where we want to “cut” and how many categories will be created.
• “Distance” (defined various ways) is measured between clusters, in order to make the splitting/merging decisions.
• Agglomerative hierarchical clustering starts with all points as clusters of size 1, and merges them until reaching the “root” of the tree

Hierarchical clustering example. Cutting vertically, at different points along the x axis, will create different numbers of clusters. Image source: Wikimedia Commons

Hierarchical clustering code example

hc_spec <- hier_clust(
  num_clusters = 3,
  linkage_method = "average"
)
hc_spec

hc_fit <- hc_spec %>%
  fit(~ body_mass_g + bill_length_mm,
    data = penguins
  )

hc_fit %>%
  summary()

hc_fit$fit %>% plot()

Using model results

• Developing familiarity with the data
• For “labeling”—applying classes to observations
  • In a subsequent step, if we treat these classes as ground truth, we could also create a classifier! (supervised method)
• For data cleaning (anomaly detection)
• For investigation of interesting cases (anomaly detection)
• For visualization (dimensionality reduction)

How to choose a ML method?

• What do you want to do? (regress, classify, reduce number of variables, cluster, find anomalies)
• What special properties do your data have? (Very skewed? Many missing values? Many covariates? Etc.)
• What approaches have appeared in literature?
• What would make for an interesting data exploration or question? (What ideas can I steal from [science journalism of] other fields?)

Recommended reading

or, looking for discipline-specific R?

Check out the Big Book of R! An online directory at https://www.bigbookofr.com/ of very many R ebooks, most of them free OER and produced by experts, organized by discipline/topic and searchable.

Look up your discipline (or some topic that interests you, e.g., time series data) and see what applications of R you can find.

Example graphic of a recent update